Sharp Detection in Pca under Correlations: All Eigenvalues Matter

Principal component analysis (PCA) is a widely used method for dimension reduction. In high dimensional data, the “signal” eigenvalues corresponding to weak principal components (PCs) do not necessarily separate from the bulk of the “noise” eigenvalues. Therefore, popular tests based on the largest eigenvalue have little power to detect weak PCs. In the special case of the spiked model, certain tests asymptotically equivalent to linear spectral statistics (LSS)—averaging effects over all eigenvalues—were recently shown to achieve some power. We consider a nonparametric “local alternatives” generalization of the spiked model to the setting of Marchenko and Pastur (1967). This allows a general correlation structure even under the null hypothesis of no significant PCs. We develop new tests to detect weak PCs in this model. We show using the CLT for LSS that the optimal LSS satisfy a Fredholm integral equation of the first kind. We develop algorithms to solve it, building on our recent method for computing the limit empirical spectrum. Our analysis relies on the new concept of the weak derivative of the Marchenko-Pastur map of eigenvalues, which also leads to a new perspective on phase transitions in spiked models.